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Introduction 


■ This presentation describes a system developed to extract metrics from an 
existing status report 

- Provides in-depth data for Evidence-Based Assurance (EBA) and milestone analysis etc., with 
no increase in analysts' workload 

■ Initial analysis toward a traditional IT approach 

- Thoughts behind the change in direction 

■ Describe the BIWAR report, and how it was already structured to support 
post-processing 

- Styles, formatting, required identifying elements 

■ Describe patterns processed by the software 

- Sets of key phrases that point to data 

■ Flexibility through configurability 

- Maintain search phrases in external configuration files 

■ Additional ideas using similar techniques 
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The problem 


■ The ISS team needed to provide accessible metric data for EBA, while 
limiting impact to on-going IV&V review and Independent Analysis (IA) work 

■ The BIWAR had already been identified as a source for EBA data, but it was 
not really accessible for after-the-fact analysis 
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Initial analysis and prototyping 


■ ISS Innovation team was asked to provide analysis/prototyping support for 
a solution 

- One of the first tasks undertaken 

■ Initial analysis generated an Operational Concept and data entry screen 

- Defined a process to move the inputs from analyst to lead to report, including modification 
tracking 

- Defined a template document structure to "mail merge" with input data to generate the 
report 

o Created a proof of concept to show the idea is feasible 

- Generated a main data entry panel 

o Selection/Entry of "top-level" data for report generation 
o Did not include the lower-level metric data needed for each analysis record 
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Initial analysis and prototyping (cont'd) 


■ Example template document 

BIWAR 

[! section CCS_R14 !] 

CCS R14 

[! section CCS_R14 milestone !] 

During this period, IV&V performed the following milestone reviews for CCS R14. 
[! start user data area !] 

[! end user data area !] 

[! end_section CCS_R14 milestone !] 

[! start user data area !] 

[! end user data area !] 

[! end_section CCS_R14 !] 
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Initial analysis and prototyping (cont'd) 


■ State machine 

- Analyst inputs, to Team Lead (TL) to report 
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Initial analysis and prototyping (cont'd) 
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■ 



■ Although the work had progressed, it was far from complete 

- The Operational Concept had defined a fairly complex state machine to manage inputs as 
they move from analyst to lead to report 

o No work on that had started 

- The data entry screen was display only 

o No implementation behind it 

- Only "top-level" data defined 

o Specific metric data was yet to be defined 
o Another entry screen is needed 

o The current accepted data scheme has 14 columns that were not included in the initial data entry 


screen 
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■ 



■ The next step was to identify the necessary metric data and define how the 
analyst will enter it 

- Define a data dictionary 

- Define a data entry screen to capture that info 

- Realized the primary source for discovering what data to include was the report we already 


■ Began to recognize usability issues 

- Analysis results entered on a data entry form vs. using Microsoft Word 

o More natural/convenient for the results to be entered in Word 

- Additional work to maintain the data in a new tool 

- Managing yet another work flow 

■ Something I had written in the Con Ops kept coming back to me 

- The system should be able to reverse itself based on the template 
So it can go backward from the final report to individual data 
Very useful for testing if could be done 


Could the existing report be mined for data, which is in 
turn used to generate the metrics database? 


generate 
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■ 



■ The ISS team has been generating BIWARs basically since the project kicked 

off 

- The team is already proficient at providing the data in the periodic report 

- BIWAR is accepted as a source of EBA data and Assurance statements for the ISS team 

■ Each analyst is responsible for providing inputs 

- Funnel up to team leads for final filtering and combination of individual analyst reports into 
the team report 

■ Analysts have guidelines defining what must be reported as well as 

generally accepted formatting standards. 

- If inputs are non-compliant, leads can either update inputs to fit, or kick the report back to 
the analyst for updates. 

- The team's administrative assistant enforces rules on the paragraph styles as part of 
producing the final complete ISS report. 

■ NASA PM is the primary customer for the report 

- With each report a meeting is held to ensure the information in the report is accurate and 
the analyses performed are understood 
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■ 



■ Report contains many metrics used as evidence 

- CSCI revision 

o Assembly Stage 

- IPR and Milestone reviews 

o Review type 
o Artifact reviewed 
o Analyst commentary 
o Related change document 
o Issue counts, dispositions, closure commentary 

- Contributions, Value Statements, Lessons Learned 

o Reports all data mentioned above related to a significant accomplishment by IV&V 

- SCRs reviewed 

o Content list 
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■ 



■ The report has a consistent format 

- Styles are used to format the data 

o Also identifies section breaks, and content of data described in the text 

- Analysts use a consistent form 

o SCR 12345, "The SCR Title " 

o Specific names used for Milestone and IPR review types 
o References the Program review ID (RCS #, TRSR #, etc..) 

■ Styles can drive a state machine 

■ Algorithms can identify metric data based on the phrasing of the text 
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■ 



■ Example BIWAR entry 
CCS R14 

IV&V performed a design/code review for I/O changes related to SCR 41943, "CCS 
changes for ICU Modem Ku-Band LOS/AOS Issues". IV&V reviewed code and design 
document changes for this update and found a somewhat confusing or misleading 
statement in the TLDD related to the Last Loss of Signal (LOS) Override Command. 
IV&V generated one accepted issue for the review to request an update to clarify 
that statement and bring it into agreement with the actual implementation. 

■ Capture the CSCI (CCS R14) 

- Capture that the ICU CSCI was also mentioned 

■ Capture that this was a design and code review 

■ Capture the SCR number 

■ Capture the number of issues generated for the review 
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■ 



■ Example BIWAR entry 

ICU R3 

IV&V performed a requirement review for RCS 0108 of SCR 41258 " Add 1553 Telemetry for 
USCG39FC3100L Set Function command ” . IV&V noted that a requirement changed to handle 
the Command_to_Perform_a_Modem_Loopback_Test set part of the new telemetry at the 
start of command processing and set the rest after the Loopback test completes. There could 
be an extended period of time between setting the telemetry values, which would make them 
difficult to track by operators. IV&V generated one issue, PIMS 3660-00002, to report that 
problem. The issue was accepted and requirements were reorganized to match the planned 
implementation. 

■ Capture the ICU R3 CSCI 

■ Capture this was a requirement review 

■ Capture the RCS number 

■ Capture the SCR number 

■ Capture the number of issues generated 
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■ 



■ A prototype was developed to show that the concept is feasible 

- Reviewed and approved by NASA PM for ISS 

■ Multiple revisions were made as updated reports were delivered 

- Adding new key phrases and search phrases to capture diverging phrasing noted in 
successive reports 

- Adding new features for capturing different data as suggested by the Team leads on review. 

■ Input report is searched paragraph by paragraph for key phrases 

- Related data is captured 

- Simple text search, regular expressions were not needed 

■ Output data generated in html tabular format 

- Brute force for now, no CSS 

■ One HTML-formatted output file is generated per report 
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■ 



■ HTML output designed to be opened in Excel for maintenance 

- Provides extensive data query capabilities with no additional IV&V effort required 

- Allows for manual edits if needed, along with built in commenting so changes are tracked 

■ Admin personnel can easily merge approved reports into the historical 
database 

- Provides a measure of version control due to having a single "choke point" 
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Prototype 2 (cont'd) 


[X] ISS BIWAR examples.html - Microsoft Excel 
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Prototype 2 (cont'd) 


[X] ISS BIWAR examples.html - Microsoft Excel 

Home Insert Page Layout Formulas Data Review View Developer Add-Ins 
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■ 



■ Multiple revisions were made as updated reports are delivered 

- Adding new key phrases and search phrases to capture diverging phraseology noted in 
successive reports 

- Adding new features for capturing different data as suggested by the Team leads on review. 

- Configurability is KEY for the success of this application 

■ Configuration data maintained in an external file 

- Application configuration data 

- Key phrases and search patterns 

■ Easy to add functionality 

- File is tab formatted, plain text, can be modified using any text editor 

- Or can be managed in Excel 

o Set up ranges and selection lists to validate the configuration names 

o Can either export as Tab-delimited file or copy/paste the row data to create the "config.dat" file 
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Built-In Configurability (cont'd) 
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■ 



■ Analysis identified unique word structures that could be used to locate data 
in the report 

- Self-identifying - key phrases that are themselves data 

o CSCIs (CCS R13, CCS R14, C2V2, ICU R2, ICU R3, NCS R4, etc..) 
o Milestones (PDR, CDR, TRR, TDR, STRR, etc..) 

- Call-response - key phrases that return a specific value 

o Start/Stop parsing section names (FUNCTIONAL TEAMS = Start, PROJECT MANAGEMENT AND CONTROL = 
Stop) 

o Paragraph types (biwar 1 = 2, biwar 2 = 3, biwar 3 = 4, etc) 
o Drop phrases (No significant activity, Nothing to report, None to report, etc..) 
o Activity names (Significant Activities, Independent Analysis, Contributions, etc.) 

- Lead-follow - key phrases that signify data to follow 

o Limits word span between leader phrase and follower data and ensures they do not span sentences 
o Artifact names/IDs (SCR, RCS, Test package, etc..) 
o IV&V generated SCRs 
o IV&V Risk #s 

- Lead-follow-trail - pairs of phrases that work in tandem to ID metric data by surrounding it 

o Limits word span between lead/trail phrases, and ensures they do not span sentences 
o Review type detectors (("IV&V participated in", "review"), etc..) 
o Issue count detectors (("IV&V generated", "new issues"), etc..) 

No code change is needed to add/change/delete a pattern because they are defined in the configuration file 
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Search text patterns (cont'd) 


■ Example Lead-trail phrases 

IV&V generated SCR 
IV&V initiated SCR 
IV&V submitted SCR 
IV&V created SCR 
IV&V Risk 
IV&V Risk# 

SCR 

SCRs 

RCS 

RCSs 

RCSes 

TUI 

TUIs 

TUI# 

TUI#s 
TUI #s 

Test package 
test packages 
test procedure 
test procedures 
TRSR 
TRSRs 
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Search text patterns (cont'd) 


Example Lead-follow-trail phrases 

IV&V generated 

accepted RID 

IV&V identified 

accepted RID 

IV&V found 

accepted RID 

IV&V had 

accepted RID 

IV&V submitted 

accepted RID 

IV&V generated 

issues 

IV&V identified 

issues 

IV&V found 

issues 

IV&V had 

issues 

IV&V submitted 

issues 

IV&V performed the following 

review 

IV&V performed 

review 

IV&V participated in 

review 

IV&V performed the following 

reviews 

IV&V performed 

reviews 

IV&V participated in 

reviews 
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■ 



■ An early challenge to the application was related to a writing style where a 
lead-in paragraph leads to subsequent detailed information 

■ There are two formats used: 

1. The subsequent paragraphs all inherit any data mined from the 
introduction, but each is independent from the other related paragraphs 

2. The introduction and subsequent paragraphs should all be treated as one 
for data mining purposes 
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■ 



■ Example Format 1: 

- IV&V performed the following post-FQT test reviews: 

o VRC #2014-05, SIF #6708 "CCSR13 PPL 51 and PPL 177 To Provide Payload Timeliner 
Command Access to Support AMO_EXPRESS". This VRC contained FQT TUI# CCS_R13- 
SIF6708.CR-01. This TUI verified the update for PPL 51-Payload Command Request and 
PPL 177-Payload Adaptation Data for SCR 41963 "PLMDM Command Requirements 
Required for AMO EXPRESS" and SCR 41920 "CCS PAD PPL to Support Payloads AMO 
EXPRESS". IV&V identified two test design discrepancies: 1) incomplete test 
assumption and constraints, 2) missing test confirmation of check safing response is 
disabled. FQT accepted and updated the STD document, as recommended. 

o VRC #2014-05, SIF #6755 "CCS R13 PPL 278- Resolve ISS Thermal Control System (ITCS) 
Range-Check Issue". This VRC contained FQT TUI# CCS-CDH. DA-04- Range-checking of 
External Functional Data. This TUI verified the update for PPL 278-Range Check Data for 
SCR 42032 "Incorrect standard out definitions will pose problems for IFHX leak 
detection algorithms once range checked". IV&V found the following discrepancies: 1) 
verification of the signal PUIs identified in SCR 42032 was not clear from the test design, 
2) test results did not lash up with the STD that verified External Functional Data Range 
Check Error Counter of the Standby or Backup MDM was not reset to zero when the 
command to clear range check latch data was sent to the Primary MDM. FQT accepted 
and corrected all issues. 
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■ 



■ Example Format 2: 

- IV&V continued code review for SCR 40918 "Implement CR 13047/13231 for 
International Docking Adapter (IDA)" and SCR 41731 "Implement CR 13231R1 for 
International Docking Adapter (IDA) in GN&C." To date, the following have been 
completed: 

o Review of the changes to two existing modules for detecting and validating new pass- 
through standard and pass-through data load commands for the VV. IV&V found no 
issues with these code changes. 

o Review of the changes to two existing modules for activating and deactivating 
transactions on busses that the IDA interfaces share with the Orbiter Interface 
Unit/Portable Computer System (OIU/PCS) interfaces. IV&V found no issues with these 
code changes. 
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■ 



■ Format 1 was handled by defining a configuration item to identify the lead- 
in paragraph by its content 

- Paragraphs starting with 

o IV&V performed the following life-cycle reviews 
o IV&V performed the following design reviews 
o IV&V performed the following code reviews 
o IV&V performed the following test reviews 
o etc. 

- Data is parsed from that paragraph and combined with data parsed from 
subsequent related paragraphs 
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■ 



■ Format 2 was handled by implementing a data structure and search 
configuration 

- Loads the Word document, parsing it by paragraph styles into blocks 

- Instead of the top-level paragraph knowing what styles it should combine with, 
the knowledge was designed into the lower-levels 

o Looks up instead of down 

- Each style is checked with the most recent prior style, and combined if needed 
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Combining paragraphs (cont'd) 
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■ 



■ Needed the ability to recognize plural key phrases and capture multiple 
following metric data items 

- Ex. "... reviewed Test packages TESTPKG1, TESTPKG2, and TESTPKG3" 

o Test packages is the key phrase 

o TESTPKG1, TESTPKG2, and TESTPKG3 are the metric data items 

- Software was changed to detect plural phrases 

- For plural forms, will accept successive identifiers up to the end of the sentence 

o Word span rules enforced 

- if identifier must be within N words of the key phrase, then multiple identifiers 
must be within N words of each other 

o Will pick up all 3 identifiers for "... reviewed Test packages TESTPKG1, TESTPKG2, and 
TESTPKG3" with a span of 2. 

o Would NOT pick up the last identifier in "... reviewed Test packages TESTPKG1, 
TESTPKG2, and the TESTPKG3" with a span of 2. 

o Discards all punctuation following the data items 
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■ 



■ NASA PM 

- Generating STRR/SMSR/other reporting metrics 

■ TASCPM 

- Tracking progress toward milestones 

■ ISS Team leads 

- Managing analyst work 

- Metrics for summary/periodic reports 

- Reviewing past analysis work to apply to similar current situations 

o For example, say that a new SCR is reporting a condition similar to a prior SCR. The IV&V database 
could be referenced to determine what analysis was performed for the earlier SCR, and use that as a 
starting point for analyzing the new SCR. 

- Collecting evidence for technical reports 

■ IV&V analysts 

- Reviewing past work (i.e. look back at STD review when doing test implementation review) 
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Uses for the data (cont'd) 


■ The range of data captured, and the inclusion of the original text, provides 
almost unlimited query capabilities through the Excel interface 

■ Gathering approximately 100 data records per month, covering all ISS IV&V 
review 
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■ 



■ Use the key phrases defined by this work to generate text reports from raw 
data 

- Combine different phrases into styles 

- Allow user to select a style to use, and fill in the missing data 

- Select to use a random style each time to avoid repetitive language 

- Data from spreadsheet to Word document, with no word-smithing needed 
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Future possibilities (cont'd) 


■ Understanding Word automation opens doors 
- Two recent examples 
1. Multi-search 

- Search multiple word documents for multiple phrases 

- Builds a "hit list" with clickable links to the document and location 

- Allows user note entry for each hit, with full load and save 
capabilities 

- Useful for tracking data items across requirements, design, test 
documentation, as just one example 

- Possible use in generating Tech References? 
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■ 



2. Input-Output data analysis 

- Extracted Input and Output data items from the SRS into a 
spreadsheet 

- Captured all data items that are Output (modified) by more than 
one function 

- Captured all data items that are an Input to one function, and an 
Output to another 

- Used to guide analysis to determine if there is cause for concern in 
any of the captured data items 

- Define tests to verify the software in combination, where the 
behavior is suspect 

- FQT only verifies functions in isolation so this is a value add by 


IV&V 



TASC 


37 


NASA IV&V International Workshop 2014 



■ 



■ Other development details: 

- The main application is built in C#, using standard Microsoft Word automation 
capabilities 

o Microsoft Word 14.0 Object Library 

- The data structure for combining paragraphs is implemented as a set of type- 
length-value (TLV) elements 

o The TLV is a new concept to ISS, recently introduced via a new project that 
uses it for configuration data 

o Learning by using 

- php was also used, to convert the TLV data from text to binary format 

o Quickest way to get from point A to point B 

- A Finite-State-Automaton (FSA) is used for the paragraph style search in the TLV 
data 

o Adapted from " More Programing Pearls " by Bentley 1988 
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